{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7c249b40",
   "metadata": {},
   "source": [
    "# Using Amazon Bedrock\n",
    "\n",
    "Amazon Bedrock is a fully managed service that makes FMs from leading AI startups and Amazon available via an API, so you can choose from a wide range of FMs to find the model that is best suited for your use case.\n",
    "\n",
    "This tutorial will show you how to use Amazon Bedrock with Ragas.\n",
    "\n",
    "1. [Metrics](#load-sample-dataset)\n",
    "2. [Testset generation](#test-data-generation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e63f667",
   "metadata": {},
   "source": [
    ":::{Note}\n",
    "this guide is for folks who are using the Amazon Bedrock endpoints. Check the [evaluation guide](../../getstarted/evaluation.md) if your using OpenAI endpoints.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f466494a",
   "metadata": {},
   "source": [
    "## Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e54b5e01",
   "metadata": {},
   "source": [
    "### Load sample dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b658e02f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Repo card metadata block was not found. Setting CardData to empty.\n"
     ]
    }
   ],
   "source": [
    "# data\n",
    "from datasets import load_dataset\n",
    "\n",
    "amnesty_qa = load_dataset(\"explodinggradients/amnesty_qa\", \"english_v2\")\n",
    "amnesty_qa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4b8a69c",
   "metadata": {},
   "source": [
    "Lets import metrics that we are going to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "f17bcf9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import (\n",
    "    context_precision,\n",
    "    faithfulness,\n",
    "    context_recall,\n",
    ")\n",
    "from ragas.metrics.critique import harmfulness\n",
    "\n",
    "# list of metrics we're going to use\n",
    "metrics = [\n",
    "    faithfulness,\n",
    "    context_recall,\n",
    "    context_precision,\n",
    "    harmfulness,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1201199",
   "metadata": {},
   "source": [
    "Now lets use the llm from Bedrock using `BedrockChat` class from Langchain. Init a new instance of `BedrockChat` with the `model_id` of the model you want to use. You will also have to change the `BedrockEmbeddings` in the evaluate function with the metrics that we use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "40406a26",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.chat_models import BedrockChat\n",
    "from langchain_community.embeddings import BedrockEmbeddings\n",
    "\n",
    "config = {\n",
    "    \"credentials_profile_name\": \"your-profile-name\",  # E.g \"default\"\n",
    "    \"region_name\": \"your-region-name\",  # E.g. \"us-east-1\"\n",
    "    \"model_id\": \"your-model-id\",  # E.g \"anthropic.claude-v2\"\n",
    "    \"model_kwargs\": {\"temperature\": 0.4},\n",
    "}\n",
    "\n",
    "bedrock_model = BedrockChat(\n",
    "    credentials_profile_name=config[\"credentials_profile_name\"],\n",
    "    region_name=config[\"region_name\"],\n",
    "    endpoint_url=f\"https://bedrock-runtime.{config['region_name']}.amazonaws.com\",\n",
    "    model_id=config[\"model_id\"],\n",
    "    model_kwargs=config[\"model_kwargs\"],\n",
    ")\n",
    "\n",
    "# init the embeddings\n",
    "bedrock_embeddings = BedrockEmbeddings(\n",
    "    credentials_profile_name=config[\"credentials_profile_name\"],\n",
    "    region_name=config[\"region_name\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44641e41",
   "metadata": {},
   "source": [
    "Now we can use the llm and embeddings with `Bedrock` by passing it in the evaluate function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d6ecd5a",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Running the evalutation is as simple as calling evaluate on the `Dataset` with the metrics of your choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5d0aac10",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    eval: Dataset({\n",
       "        features: ['question', 'ground_truth', 'answer', 'contexts'],\n",
       "        num_rows: 20\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amnesty_qa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "22eb6f97",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluating: 100%|██████████| 12/12 [01:05<00:00,  5.48s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'faithfulness': 0.6250, 'context_recall': 1.0000, 'context_precision': 1.0000, 'harmfulness': 0.0000}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ragas import evaluate\n",
    "import nest_asyncio  # CHECK NOTES\n",
    "\n",
    "# NOTES: Only used when running on a jupyter notebook, otherwise comment or remove this function.\n",
    "nest_asyncio.apply()\n",
    "\n",
    "result = evaluate(\n",
    "    amnesty_qa[\"eval\"].select(range(3)),\n",
    "    metrics=metrics,\n",
    "    llm=bedrock_model,\n",
    "    embeddings=bedrock_embeddings,\n",
    ")\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2dc0ec2",
   "metadata": {},
   "source": [
    "and there you have the it, all the scores you need.\n",
    "\n",
    "now if we want to dig into the results and figure out examples where your pipeline performed worse or really good you can easily convert it into a pandas array and use your standard analytics tools too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8686bf53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>answer</th>\n",
       "      <th>contexts</th>\n",
       "      <th>faithfulness</th>\n",
       "      <th>context_recall</th>\n",
       "      <th>context_precision</th>\n",
       "      <th>harmfulness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the global implications of the USA Su...</td>\n",
       "      <td>[The global implications of the USA Supreme Co...</td>\n",
       "      <td>The global implications of the USA Supreme Cou...</td>\n",
       "      <td>[- In 2022, the USA Supreme Court handed down ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Which companies are the main contributors to G...</td>\n",
       "      <td>[According to the Carbon Majors database, the ...</td>\n",
       "      <td>According to the Carbon Majors database, the m...</td>\n",
       "      <td>[- Fossil fuel companies, whether state or pri...</td>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Which private companies in the Americas are th...</td>\n",
       "      <td>[The largest private companies in the Americas...</td>\n",
       "      <td>According to the Carbon Majors database, the l...</td>\n",
       "      <td>[The private companies responsible for the mos...</td>\n",
       "      <td>0.25</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What are the global implications of the USA Su...   \n",
       "1  Which companies are the main contributors to G...   \n",
       "2  Which private companies in the Americas are th...   \n",
       "\n",
       "                                       ground_truth  \\\n",
       "0  [The global implications of the USA Supreme Co...   \n",
       "1  [According to the Carbon Majors database, the ...   \n",
       "2  [The largest private companies in the Americas...   \n",
       "\n",
       "                                              answer  \\\n",
       "0  The global implications of the USA Supreme Cou...   \n",
       "1  According to the Carbon Majors database, the m...   \n",
       "2  According to the Carbon Majors database, the l...   \n",
       "\n",
       "                                            contexts  faithfulness  \\\n",
       "0  [- In 2022, the USA Supreme Court handed down ...           NaN   \n",
       "1  [- Fossil fuel companies, whether state or pri...          0.00   \n",
       "2  [The private companies responsible for the mos...          0.25   \n",
       "\n",
       "   context_recall  context_precision  harmfulness  \n",
       "0             NaN                1.0          NaN  \n",
       "1             NaN                NaN          NaN  \n",
       "2             1.0                NaN          0.0  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = result.to_pandas()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b133aff0",
   "metadata": {},
   "source": [
    "## Test Data Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c7192f2",
   "metadata": {},
   "source": [
    "Load the documents using desired dataloader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "529266ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import UnstructuredURLLoader\n",
    "\n",
    "urls = [\n",
    "    \"https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023\",\n",
    "    \"https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023\",\n",
    "]\n",
    "loader = UnstructuredURLLoader(urls=urls)\n",
    "documents = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87587749",
   "metadata": {},
   "source": [
    "now we have documents created in the form of langchain `Document`\n",
    "Next step is to wrap the embedding and llm model into ragas schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d5eaed2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.llms import LangchainLLMWrapper\n",
    "from ragas.embeddings.base import LangchainEmbeddingsWrapper\n",
    "\n",
    "bedrock_model = LangchainLLMWrapper(bedrock_model)\n",
    "bedrock_embeddings = LangchainEmbeddingsWrapper(bedrock_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7d17468",
   "metadata": {},
   "source": [
    "Next Step is to create chunks from the documents  and store the chunks `InMemoryDocumentStore`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e717c13",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.testset.extractor import KeyphraseExtractor\n",
    "from langchain.text_splitter import TokenTextSplitter\n",
    "from ragas.testset.docstore import InMemoryDocumentStore\n",
    "\n",
    "splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)\n",
    "keyphrase_extractor = KeyphraseExtractor(llm=bedrock_model)\n",
    "\n",
    "docstore = InMemoryDocumentStore(\n",
    "    splitter=splitter,\n",
    "    embeddings=bedrock_embeddings,\n",
    "    extractor=keyphrase_extractor,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7773f4b5",
   "metadata": {},
   "source": [
    "Initializing `TestsetGenerator` with required arguments and generating data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "495ff805",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.testset import TestsetGenerator\n",
    "from ragas.testset.evolutions import simple, reasoning, multi_context\n",
    "\n",
    "test_generator = TestsetGenerator(\n",
    "    generator_llm=bedrock_model,\n",
    "    critic_llm=bedrock_model,\n",
    "    embeddings=bedrock_embeddings,\n",
    "    docstore=docstore,\n",
    ")\n",
    "\n",
    "distributions = {simple: 0.5, reasoning: 0.25, multi_context: 0.25}\n",
    "\n",
    "# use generator.generate_with_llamaindex_docs if you use llama-index as document loader\n",
    "testset = test_generator.generate_with_langchain_docs(\n",
    "    documents=documents, test_size=10, distributions=distributions\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a80046b",
   "metadata": {},
   "source": [
    "Export the results into pandas¶"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b4633c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = testset.to_pandas()\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f668fce1",
   "metadata": {},
   "source": [
    "And thats it!\n",
    "\n",
    "if you have any suggestion/feedbacks/things your not happy about, please do share it in the [issue section](https://github.com/explodinggradients/ragas/issues). We love hearing from you 😁"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
